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Organized physical education is of vital importance to the health and 
welfare of Americans of all ages and to the U. S. society as a whole* This 
discipline carries a major responsibility for helping citizens to develop and 
maintain fitness, vigor, motor skills , and heal th-promo ting habits of die t , 
exercise, and recreation. Such outcomes are crucial, considering the high 
level of heart disease, stress, and other maladies in the U. S. It is appro- 
priate that the physical education field advance its contributions. As John 
Taylor said, networking is one way to do that. Another way is to study the 
physical education needs of the field's constituents, evaluate its programs 
and priorities, and strengthen its policies and services. 

I am glad to join with this AERA special interest group to explore how 
physical educators can improve their evaluations of policies and programs. My 
assignment in the symposium is to identify methodological issues in policy 
studies and provide advice for addressing these issues. 

In preparation for the symposium, I sought to review as much relevant 
material as possible. I wrote to the physical education specialists in ea^h 
state and territorial department of education, to the major national organiza- 
tions concerned with the field, and to about a dozen of the field 1 s national 
leaders. I informed them of my assignment and asked for assistance in identi- 
fying .ecent and current studies concerned with policies in physical educa- 
tir~. I also obtained the results from two library searches for the same 
fiatei .1. 

The haul from these information retrieval efforts was minuscule. Mainly, 
it consisted of a few position papers, quite a few apologetic letters, and 
only four policy studies. These included two assessments of fitness, one 
study of physical education requirements in higher education, and one study of 
the workloads of physical education instructors in higher education. Most 
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respondents said they knew of no policy studies concerned with physical 
education. 

To supplement the information obtained from these efforts, I have 
reviewed the report of a 1976 International Congress on evaluation in physical 
education. The Congress was held in Finland and included representatives from 
seventeen countries. I gave the keynote address and reviewed the other 33 
papers presented during the 5-day conference. I was impressed with the extent 
and quality of the evaluation work reported. 

I am puzzled why ray 19^7 search for similar studies turned up only four 
studies. Perhaps the key issue in this symposium is not how to improve the 
quality of policy studies in physical education, but how to increase the 
number of such studies. Or, maybe I need to improve ray abilities to locate 
and retrieve study reports. 

In aay case, I have prepared this four-part paper using the relevant 
information that I could find. In the first part of the paper, I define what 
I mean by the tarra ! 'policy." In the second part I describe two basic types of 
policy studies. In the third part I define four major issues in planning and 
conducting studies. In the fourth and final part, I discuss how the Standards 
for Evaluations of Educational Programs, Projects, and Materials (Joint Com- 
mittee, 1981) can be applied in planning ?nd conducting policy studies in 
physical education. 
Definition of Policy 

Thomas Green (1982) has argued that, in order to cover the range of 
ordinary usage, the term policy must command a very broad interpretation. 
Accordingly, policy can mean priorities, goals, preset decisions, rules for 
decision making, guides to practice, and, in general, predispositions to 
respond in certain ways to common situations. 



Given this broad definition, a host of policies in physical education and 
sport come to mind. To help us think together in this area, let me provide a 
few examples. 

One is the formal public prescription by a state or national agency (for 
example, how many hours of physical education instruction must be offered 
weekly to each student or to a certain group of students, such as those with 
handicaps). Other policies are a state's prescribed academic requirements for 
certifying teachers of physical education, also their rules governing the 
participation of transfer students in organized sports. Another example is 
the strategic plan of a professional association of physical educators (e.g., 
the positions to be argued and the approach to be followed in lobbying for or 
against given legislation). Another example policy is the section in a school 
district's policy handbook that prescribes the amount, content, and sequence 
of physical education offerings. Another policy is seen in the rules of thumb 
a coach uses to prepare his team for athletic competitions (for example, play 
the next game before discussing the one to follow it; or, in preparing for a 
Friday game, practice hard on Wednesday and light on Thursday). As a final 
example, an athletic conference's unwritten agreement that basketball is a 
contact sport and that referees should call few fouls is a policy familiar to 
Big 10 basketball fans. 

These are all predispositions to act in certain ways in given types of 
situations, and they are therefore policies. It doesn't matter whether they 
are written or officially adopted. If they are used repeatedly to guide 
actions, they are policies. 

Moreover, all such policies are potential topics for evaluation. They 
help determine what services are provided and how they are delivered. The 
policies may be effective or of no consequence. They may have side effects 



that are positive or harmful. They may be equitable or discriminatory. They 
may be sound on paper but flawed in practice. They mr y be very good and 
beneficial and only in need of validation. For these and related reasons, 
government agencies, professional associations, school boards, teachers, 
coaches, and others need to obtain evaluative feedback they can use to guide 
the f onnulati on , implementation , and improvement of the policies and guide 
their work. 

Types of Policy Studies 

So, what kinds of policy studies do these groups need? 

Basically, I believe they need feedback from tvo main types of evaluative 
studies. Each type can be broken down into subtypes; but lor the purpose of 
this presentation, it is sufficient to look only at the major division. 

The first type includes studies to assist in the formulation of policies. 
Examples include needs assessments aimed at revealing ieficiencies (for 
example, in the fitnjss of mentally handicapped children), and evaluations of 
program plans (for example, a critical comparison of a given state 1 s plan for 
physical education with that of a country where fitness test results are 
better). Such studies can assist in formulating goals, selecting plans, 
allocating resources, assigning responsibilities, and motivating public and 
government support. The aim of all these studies is to assist in formulating 
or revising policies. 

This type of study seems about as common as morel mushrooms in Michigan 
in July. There may be some around; but, if so, they are very hard to find. 
There were only a few of these studies in the Finland report; and, in a more 
general context, Gerald Bracey (1982) reported that studies that influence 
policy are rare throughout education. Nevertheless, new policies do get 
formulated and adopted. More often than not, the policies are generated 



through a political process or are personally determined. Rarely do research 
or evaluation studies guide the formulation of the policies. One way physical 
educators could strengthen their policy development work is to conduct more 
studies aimed at guiding and informing the policy development process. 

The second type of policy study is more prevalent in physical education. 
It includes studies to evaluate the implementation and effectiveness of given 
policies. These studies come into play after a policy has been enacted. They 
examine the implementation and effectiveness of the policy. They provide 
feedback to assist in program management, information for accountability 
reports, and stimulation for revising policies. Sometimes these studies are 
called process and product evaluations. Most of the studies reported at the 
Finland conference were of this second type. 
Methodological Issues in Policy Studies 

Now that you have some idea of how I define policy and the kinds of 
policy studies I have in mind, I turn to tpe main topic of this paper. It 
concerns key methodological issues in policy studies. In other words, if you 
are planning a policy study, what issues should you anticipate and what mis- 
takes should you avoid? 

The most comprehensive response to these questions is embodied in the 
Standards for Evaluations of Educational Programs, Projects, and Materials 
(Joint Committee, 1981). These standards were developed by a 17-member com- 
mittee representing twelve professional societies concerned with evaluations 
in education. The Standards are appropriate for application to policy 
studies, because, as illustrated above, policy studies essentially are evalu- 
ations. In developing the standards, the Joint Committee identified and 
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addressed four general issues in evaluation work. The »e issues concern the 
utility, feasibility, propriety, and accuracy of evaluation studies. I vill 
comment briefly about each of the issues. 

The problem regarding utility is that many evaluations are designed, 
conducted, and reported so that they are of little interest or use to any 
particular client group. The Joint Committee identified many mistakes that 
can work against utility. A study might be focused on questions that are of 
no particular interest to anyone but the evaluator. The evaluators might fail 
to earn the confidence of the intended audience. The obtained information 
might be too general or too narrow. The basis for judgments and recommenda- 
tions might be obscure or unacceptable to the audience. The reports might be 
unclear, delivered too late, or not disseminated at all. Finally, the inves- 
tigator might be unwilling or unable to assist the client to apply the find- 
ings* All such mistakes and shortcomings limi«" or impair the utility of an 
evaluation. They must be avoided if evaluations are to contribute benefi- 
cially to the development and implementation of policies. 

At the outset, if vou know that no one would use the results from a 
proposed study, then you should not conduct it. A study not worth doing isn't 
worth doing well. 

The second major issue addressed by the Joint Committee Standards con- 
cerns the feasibility of study plans and procedures. This issue is that 
evaluators often fail to consider the constraints and hazards in the evalu- 
ation situation. Consequently, their study mignt be seriously hampered or 
compromised, or it might have to be abandoned in mid-stream. One common 
problem is the inappropriate use of experimental design. Often, the require- 
ments of such designs cannot be met in field settings; or, if the requirements 
ate met, the findings are necessarily too narrow to address the client's 
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que itions. Another problem is the failure to recognize and address pertinent 
political pitfalls. The evaluation might be resisted or even sabotaged by one 
or more groups that fear its consequences. A final feasibility problem is 
that studies SDmetimes waste resources, either through inefficiency or by 
collecting useless information. 

Next, we look at the third issue: propriety. The core failings here are 
that studies sometimes are conducted illegally or unethically. If there is no 
written confirmation, working agreements might be abrogated. If conflicts of 
interest are not identified and controlled, decisions affecting the study or 
write-up of results might be compromised* In the face of embarrassing find- 
ings, clients might censor, distort, or withhold reports, sometimes with the 
complicity of the evaluator. The rights of participants in the study might be 
violated, especially by failing to obtain their informed consent before col- 
lecting information *;hat pertains to their personal characteristics or perfor- 
mance. Reports are sometimes biased in their presentation cf strengths or 
weaknesses. And sometimes evaluation funds are misappropriated. 

The fourth and final issue is accuracy. It involves problems that com- 
bine to produce erroneous findings. The accuracy of results is often compro- 
mised in a number of different ways. The evaluator and client might fail to 
clearly define the policy issue or program to be studied. The report might 
overlook contextual factors that influenced the findings. Changes in the 
study purposes and proced'ires might be made for good reasons, but not properly 
documented and considered in interpretations of findings. The samples of 
respondents might be unrepresentative. The data-gathering procedures and 
instruments might lack sufficient validity, reliability, and objectivity. If 
the procedures for coding, storing, and retrieving information are loose, data 
may be lost or improperly coded. The results might be analyzed with 
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inappropriate procedures. And, the evaluator might go far beyond the bounds 
of the available findings in generating conclusions and recommendations. Many 
of these mistakes are unde^ whe control of the evaluator. On the other hand, 
evaluator© lack the controls that are customary in laboratory research and 
must, therefore, make special efforts to overcome the problems of field 
studies. Most emphatically, evaluators must present their clients with an 
honest account of the limitations and shortcomings of their studies. 

This concludes ray account of the issues of utility, feasibility, propri- 
ety, and accuracy. Clearly, theFe four issues encompass a host of problems 
that threaten the success of policy studies. I turn next to the Joint Commit- 
tee Standards , which were constructed to help evaluators address these issues. 
Applying the Joint Committee Standards 

The 30 standards are divided according to the four issues. There are 
eight utility standards, three feasibility standards, eight propriety stan- 
dards, and eleven accuracy standards. 

E^ch standard presents and explicates an objective to be achieved in 
evaluation work, e.g., to obtain reliable measurements. The content of the 
standard explains what criteria must be satisfied in order to achieve the 
objective, what common errors should be avoided, and what procedures can be of 
assistance. Each standard also incJudes an illustration of how it applies in 
a sample evaluation. 

I will comment about four of the standards that are particularly helpful 
in planning policy studies. That's all 1 can discuss in the available time; 
but, hopefully, you'll get a general idea of how the entire set of standards 
can be used to plan policy studies. 

First, I wane to discuss the very first standard in the Utility stu. It 
is the Audience Identification standard. This standard directs the evaluator 
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to identify and rank order the audiences for the study, then to ask them to 
identify the most important study questions. Moreover, the evaluator might 
engage an advisory group representing the different audiences to provide 
reactions and advice throughout the course of the study. By using the advice 
from this standard, evaluators are helped to focus their studies on questions 
that are important to a defined client group and to meaningfully engage repre- 
sentatives of that group in conducting the study. These are crucially impor- 
tant steps towards conducting the right study and conducting it so that its 
results wi-1 be valued and used, not only by the investigator, but especially 
by the decision makers. 

The second standard I want to consider is the first one in the Accuracy 
set. It is the Object Identif ica:ion standard. It directs the evaluator to 
carefully describe the policy or program to be studied so that later the 
client group will be able to gain a clear idea of what was evaluated, its 
different forms, and how it changed during the study. Basically, this stan- 
dard counsels evaluators to describe the object of the study in depth, from 
various perspectives, and at different points in time. 

Now let's lo^K at the second standard in the Feasibility set* It is the 
Political Viability standard. It advises the evaluator to anticipate the 
different positions of various interest groups, so that their cooperation may 
be obtained, and so that possible attempts by any of these groups to curtail 
the evaluation or to bias or misapply the results can be averted or counter- 
acted. When planning a study, the evaluator should ask the relevant interest 
groups to provide their perspectives and advice. Also, before agreeing to 
conai'Ct an evaluation, the evaluator should make sure that it is politically 
viable. The evaluator should also obtain written guarantees that the evalu- 
ation team will have access to the relevant data and control over final 
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editing. As a final example, this standard advises the evaluator to provide 
the client groups with periodic progress reports so that the study's reported 
outco'iQes aren't surprises to any of the groups and so that *"hese groups may 
comment on draft reports. 

The fourth and final standard I want to )ook at is the first standard in 
the Propriety set. It is the Formal Obligation standard. It directs tnt 
obligations of the formal parties to an evaluation should be agreed to in 
writing, so that these parties are obligated to adhere to all conditions of 
the agreement or formally to renegotiate it. Among the suggested contents of 
the written agreement are the audiences for study reports, the questions to be 
investigated, the d3ta gathering and analysis methods, the reporting plan, the 
work schedule, the budget, and contract termination procedures. In my experi- 
ence, it is crucial to work out an advance contract to guide the work. proper 
contracts can forestall a whole host of deputes that may arise during the 
study. 

This concludes my discussion of the standards. I wil 1 "lose by summariz- 
ing what I have said. 
Closing 

All professionals in physical education are involved in setting policies 
to guide their services. These educators need evaluation studies that assist 
them to both set policies and examine the implementation and outcomes of the 
policies . 

Four general issues essentially determine the success or failure of 
policy studies. These are the issues of utility, feasibility, propriety, and 
accuracy. The Joint Committee Standards were developed precisely to help 
evaluators address these issues. 




If you have not already done so, I recommend that you obtain a copy or 
the Standards . I am confident that you will find them to be useful for plan- 
ning and conducting policy evaluations. 
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